Mobile devices are rapidly evolving into all-purpose mobile personal computing machines. The style and presentation of the content displayed on mobile devices is increasingly becoming important and is a major differentiating aspect among the wide array of mobile devices available in the market. In this regard, a crucial component of mobile device presentation is the physical characteristics of its display.
Currently, displays in mobile devices are differentiated based on their resolution, brightness, and the number of colors that the display supports. For instance, several mobile phones in the market come with displays having 262,000 colors (e.g., Motorola V3) and a resolution of 640×200 (e.g., Nokia 9300). Rapid increases in technology are constantly improving the aforementioned aspects of mobile device display. However, all the above-mentioned displays are limited in a very fundamental way, namely these displays are all two-dimensional displays that are incapable of conveying stereo parallax to the human eye. This is because both human eyes typically see the same image on the two-dimensional display. As a result, a user does not perceive three-dimensional information due to the lack of a perception of depth.
Stated another way, the human visual system captures the world using two eyes. Since the two eyes are located at spatially distinct positions, the input to each eye is different. The human brain makes use of this difference while processing the input from the two eyes and derives a perception of depth. When human eyes observe a two-dimensional display, both eyes view the exact same image and hence the brain cannot perceive much depth even if the image on the display is that of a three-dimensional object. The lack of perception by human eyes to perceive three-dimensional information on two-dimensional displays that are incapable of conveying stereo parallax is a problem that has confounded the industry for a long time. Various stereoscopic technologies have evolved over the years in order to counter this problem. The goal of stereoscopic technologies is to provide different input to both eyes. Some of these stereoscopic technologies include active stereo systems such as shutter glasses and head-mounted displays. Stereoscopic technologies such as active stereo systems, have not made noticeable inroads into the consumer electronics industry due to significant shortcomings in the technology. For instance, as known to those skilled in the art, users typically consider stereo glasses to be intrusive and inconvenient to use while head-mounted displays are often considered to be too big and bulky for use. In contrast to active stereo systems, passive stereo systems do not require any such electronic eyewear. Passive stereo systems include simple stereo glasses such as anaglyphic glasses and polarized glasses, as well as autostereoscopic displays such as holographic displays, parallax barrier displays, and lenticular displays.
Autostereoscopic displays are often referred to as a special kind of passive stereo system which provides a different approach to stereoscopic display. Unlike active stereo systems, autostereoscopic displays have made steady inroads in consumer electronics. Autostereoscopic displays are fast gaining ground as the preferred stereoscopic display largely because of the non-intrusive nature of their use (e.g., a user is not required put on any eyewear) and also because the prices of these displays continue to decrease due to rapid technological advancements. Conventional displays are based on the principle of lighting a pixel by a color that is then diffused in all directions. As a consequence, the pixel appears to have the same color from most viewing angles. Autostereoscopic displays are based on a radically different principle. For instance, in autostereoscopic displays a pixel emits different light rays in different view angles/zones. This property of an autostereoscopic display allows applications such as, user interface widgets, games, maps, calendars, address books, advertisements, web browsers, file browsers, and stereo image and viewers, to emit a different image to the left and the right eye and to convey a perception of depth to a viewer such as a user.
As fully appreciated by those skilled in the art, autostereoscopic displays can be categorized into three categories such as: 1) two-view displays, 2) head-tracked two-view displays, and 3) multi-view displays. The principles of a two-view display are illustrated in FIG. 1. As shown in FIG. 1, a two-view display ensures that each of its display elements sends out separate light towards the left and the right eye. While there is a specific ‘sweet spot’ where stereovision is best available, two-view displays typically provide leeway for a larger view-zone where the stereo image pairs can be viewed. On the other hand, when a user steps out of a view zone then he/she may not perceive a meaningful stereo effect. This lack of perception of a meaningful stereo effect may typically be countered by a second category autostereoscopic display, namely a head-tracked two-zone stereo display. The main basic concept of a head-tracked two-zone stereo display is to track the relative position of both eyes with respect to the display using sensors such as 3D trackers. A computer uses the tracked information to determine the appropriate images for the left and the right eye. As known to those skilled in the art, a head-tracked two-zone stereo display does not provide a robust technique for viewing 3D images and head-tracked two-zone stereo displays do not allow multiple viewers to view the same display.
The third category of autostereoscopic displays, namely multi-view displays will now be discussed. Multi-view displays are similar in principle to two-view displays except that each display element of the display sends out a different light ray in more than two directions. If these light rays are coordinated to represent the underlying scene, then stereovision can be perceived in each of the view zones. FIG. 2 shows an example of a three-view display having four viewpoints. Since there are multiple viewpoints (i.e., four) in the three-view display of FIG. 2 whenever the left and the right eye of a user are in two different view zones, the user is still able to perceive stereovision.
One of the earliest forms of autostereoscopic displays are the holographic displays. Holographic displays function through the interference of two light waves, namely a reference wave and a representative wave. The representative wave is a complex wave that is computed by a processor from the shape of the object to be displayed. The two light waves are projected to interfere in specific patterns using a fringe display. This special interference produces a diffraction pattern that resembles the image to be displayed. Amongst all the autostereoscopic display technologies currently available, holographic displays offer the most potential to be a truly volumetric display since they are not limited by resolution and are naturally three-dimensional displays. Nevertheless, holographic displays face significant challenges in terms of the amount of computation that they require and the amount of data that they need to process. Additionally, holographic display technology is still in its infancy and has limited color and resolution capabilities. Due to the challenges posed by holographic displays, these displays are considered to be in the experimental stages of evolution.
Consideration will now be given to another type of autostereoscopic display, namely a parallax display. Parallax displays are much older than holographic displays and they are more popular than holographic displays since they are cheap and are typically considered to be more robust than holographic displays. As shown in FIG. 3, parallax barrier displays have tiny opaque barriers in front of a display. The display behind the barrier displays two sets of images that are uniformly juxtaposed (i.e., interlaced) along their columns. The barriers are designed to occlude the outgoing light from the display in such a way that each eye sees only one image and does not see the pixels of the other image. In other words, a left image and a right image are interlaced so that the user may view the image on the parallax barrier display, as illustrated in FIG. 4. If the two images presented on the display correspond to the views of a scene as seen from the left and the right eyes respectively, then the user perceives a stereoscopic sight. Multi-view parallax barriers are currently in existence and they operate under a similar principle to that of a single zone parallax barrier display. For instance, with respect to an N-view parallax display, N+1 images are uniformly interlaced along their columns and are presented on the display that stands behind the parallax barrier. There are several parallax barrier displays currently in the market, one of which is manufactured by Sharp Electronics.
Another type of autostereoscopic display, namely a lenticular display will now be considered. As shown in FIG. 5, lenticular displays achieve the effects of parallax barrier displays by using optical lens placed in front of the display. With respect to a two-view lenticular display, there are two pixels of the display for each individual lens. The lenses are designed so that the left eye only sees the pixels corresponding to the left image while the right eye sees the corresponding pixels of the right image. In practice, the lenticular lenses of a lenticular display are long cylindrical lenses in front of the display. The lenticular display framework of FIG. 5 can be further expanded to multi-view lenticular displays by using more than two pixels under one lenticular lens. Both the parallax (discussed above and illustrated in FIGS. 3 and 4 respectively) and lenticular displays are somewhat limited in that these displays only offer horizontal stereovision. That is to say, there are no vertically aligned view zones provided by the parallax and lenticular displays of FIGS. 3 and 5. As a consequence, if a user moves his/her head up and down, then he/she is likely to see the same image and, thus does not perceive a meaningful stereoscopic effect. A significant advantage of lenticular displays over parallax barrier displays is that in addition to horizontal stereo, lenticular displays can offer vertical stereo effects. For instance, this can be achieved by modifying the lenticular display of FIG. 5 such that it uses a two-dimensional array of hemispherical (or near hemispherical) lens as opposed to the 1D (dimensional) array of long cylindrical lenses used for the two-view case described above.
There are several autostereoscopic displays in the market. These autostereoscopic displays include Mitsubishi LCD 4000, Sharp PC-RD-3D, Sharp LL-151D, with more products being developed from companies such as Phillips and Toshiba. Additionally, in 2003, NTT DoCoMo introduced a mobile phone (e.g., Mova SH505i) with an autostereoscopic display.
Initially, the cost of existing autostereoscopic displays was quite prohibitive. However, currently these displays are available for ordinary desktops and notebooks at an affordable price. The cost of existing autostereoscopic displays is expected to decrease significantly in the immediate future. In this regard, NTT DoCoMo introduced a mobile phone (e.g., Mova SH505i) with autostereoscopic display in 2003.
Autostereoscopic displays do not require power in excess of power required by conventional liquid crystal displays (LCDs) since autostereoscopic displays are fundamentally based on placing optical barriers or lenses in front of other types of conventional displays. Because of the benefits of autostereoscopic displays, these displays are expected to make a big impact in the mobile device market in the foreseeable future. Furthermore, the rapidly decreasing cost of autostereoscopic displays makes them an ideal new display technology for mobile devices. Autostereoscopic displays offer a generational upgrade in mobile display technology. The user interface as currently known will be upgraded by a whole new dimension to 3D (Three Dimensional). This upgrade affects a wide range of applications such as 3D video conferencing, 3D user interface, and stereoscopic games and they are expected to radically improve the presentation of mobile applications such as maps, calendars, address books, advertisements, image viewers, message viewers and the like. For example, displays employing autostereoscopic technology may enable a message viewer to display text messages using messages in 3D fonts. Similarly, displays using autostereoscopic technology may allow calendars to illustrate 3D image features such as, for example, multiple appointments using floating calendar features that come into focus as a user evinces an interest in viewing the appointment. Additionally, autostereoscopic displays may enable advertisements to include fancy 3D graphics in order to entice a user to view the advertisement. In light of the benefits of autostereoscopic displays, one can easily understand how 3D graphics rendering may be a critical component of mobile device technology.
Graphics traditionally has been driven by monoscopic displays which are typically designed for rendering from a single viewpoint. Consequently, most of the rendering algorithms and specialty graphics hardware currently available are geared towards handling a monoscopic rendering scenario. Even so, current standards and rendering hardware still provide mechanisms for rendering to stereo displays. For instance, existing approaches to rendering stereo display mainly involve a bare bone system that invokes the traditional monoscopic rendering system multiple times, namely once for each viewpoint. As such, while it is possible to render to stereoscopic systems using a monoscopic rendering system, it is very expensive to do so because of the high computational load associated with rendering multiple times for multiple viewpoints.
Consider the conventional rendering system of FIG. 6 in which rendering of three-dimensional content follows a highly parallel and deeply pipelined model. Since graphics rendering is a highly computationally intensive task, in modem computers a majority of the computations are done in a special hardware unit called a Graphics Processing Unit (GPU). Similar graphics-specific hardware exists in mobile SoCs (System-On-Chips) such as Texas Instruments (TI) Open Multimedia Applications Platform (OMAP) 2420 and mobile phones with such hardware are currently available from companies such as LG and Samsung. As referred to herein, a GPU encompasses graphics hardware on a computer and/or on a mobile device. Currently, GPUs such as, for example, Nvidia GeForceFX 7800 are designed for monoscopic rendering.
Geometric content in three-dimensional graphics is traditionally represented as a triangle mesh. A triangle mesh is a set of adjoint triangles in 3D space that share common vertices between them and is typically used to represent the area of an image. Open Graphics Library (OpenGL) is an industry standard (see, e.g., The OpenGL® Graphics System: A Specification Version 2.0, Oct. 22, 2004, the contents of which are hereby incorporate by reference in its entirety) for rendering 3D graphics content which follows the pipelined rendering model illustrated in FIG. 6. In this model, a central processing unit (CPU) 2 sends the triangles for rendering by sequentially sending information regarding the three vertices of each triangle in the mesh to the GPU 5. As understood to those skilled in the art, the triangle data can alternately be DMA (direct memory accessed)-transferred from the system or graphics memory after initiation from the CPU 2. Upon receipt of the vertices corresponding to a triangle, the GPU 5 dispatches these vertices to a vertex processing unit 4 having vertex shaders (not shown) in which per-vertex computation such as transformation and lighting are performed. The vertex shaders are programmable Multiple Instruction Multiple Data/Single Instruction Multiple Data (MIMD/SIMD) processors that can be programmed with languages such as, for example, an Open GL Shading Language (GLSL). Since there are typically multiple vertex shaders in a GPU, multiple vertices can be processed in parallel. The vertex shaders typically share a common set of input and output registers (not shown). For a vertex to be processed by a vertex shader, the attributes of a respective vertex, such as position and color, are written into input registers of a corresponding vertex shader. The vertex shader then uses this per-vertex information along with some pre-defined information to process the vertex. The processed vertex is then output to a respective output register. In like manner, each of the vertex shaders process the respective input vertices and outputs the vertices with new attributes, such as diffuse color, along the pipeline. For instance, after all three vertices of a triangle are processed they are assembled together to form a triangle by the primitive assembler 6. For each such assembled triangle, a rasterizer 8 determines the screen pixels that the triangle occupies and interpolates the three vertex attributes to determine the attributes of these pixels (e.g., diffuse color). The screen pixels are sent to a pixel processing unit 11 having pixel shaders (not shown) which operate on each of the screen pixels and compute the final color (e.g., specular plus diffuse color) and depth at a respective pixel. Subsequently, the pixels may go through a series of tests such as the stencil test, the alpha test, and the depth test, and if they pass all the tests they are written to a buffer 15 (with blending if enabled) (buffer 15 may also be referred to as original pixel buffer herein). After all the triangles are rendered, the resultant image is sent to the display device 17 where a corresponding image is displayed.
While the conventional GPU 5 of FIG. 6 is a model that fits well for monoscopic rendering, it is highly inefficient for stereoscopic rendering in which rendering for multiple viewpoints is required. This is because the vertex and pixel shaders do not have a capability to output viewpoint-dependent information such as illumination and projection. In this regard, the rendering approach of a conventional GPU is capable of achieving stereo display, but it is very costly to utilize a conventional GPU to achieve stereo display. This notion is especially true for mobile devices where most 3D applications are hard pressed to achieve rendering speeds of 15 frames per second. For example, in situations where stereoscopic displays utilize a monoscopic rendering system, 3D application speed is typically reduced at least by half, thus making such stereoscopic displays unpleasant to use. In addition to this undesirable scenario, some multi-view autostereoscopic displays might have many more viewpoints making rendering to such displays all the more slower due to the high computational load associated with rendering each viewpoint. As known to those skilled in the art, a 3D rendering framework typically serves as the core backbone of a user interface for autostereoscopic devices and if the 3D rendering framework has a high computational load, it is bound to consume mobile device resources and drain the battery life of a mobile device very quickly resulting in user dissatisfaction.
Given the above-described problems, current rendering technologies require a radical revamping to ensure that current rendering technologies are more efficient for autostereoscopic displays. While there is currently no way to overcome the fact that multiple views need to be rendered in a 3D rendering system, there is enough coherence in the computations required for rendering to multiple viewpoints which can be exploited. This is because most of the viewpoints of autostereoscopic displays are quite close spatially. Consequently, an object that is visible in one view is very likely to be visible in another view, although some parts of that object may not be visible in both views. Moreover, the illumination of the object is likely to be similar in the two views as well. Hence, the rendering computation made for rendering one viewpoint can be leveraged for rendering another viewpoint. In this regard, it may be advantageous to generate a method which leverages on the coherence of rendering for different viewpoints to reduce rendering speed, power consumption and bus bandwidth of an electronic device such as, for example, a mobile device.